Network-based navigation system having virtual drive-thru advertisements integrated with actual imagery from along a physical route

ABSTRACT

A network-based navigation system includes a user interface and a computer server system that can access a map database, an actual imagery database and an advertisement database in order to provide visual route guidance information having virtual drive-thru advertisements integrated with actual imagery from along a physical route. The user interface permits a user to submit a request for navigation information, receive visual route guidance information in response to the request and initiate a drive-thru presentation of at least a portion of a route. The computer server determines at least one route from the map database based on the request from the user and provides route guidance information to the user interface in response. The computer server also identifies actual imagery from the image database associated with the at least one route and selectively replaces at least one polygon region identified in the actual imagery associated with the at least one route with at least one advertisement from the ad database to create a drive-thru presentation with at least one virtual advertisement. At least a portion of the drive-thru presentation is presented to the user interface. In one embodiment, the computer server records a drive-thru charge for the virtual advertisements present in the portion of the drive-thru presentation provided to the user interface.

PRIORITY APPLICATION

This application is a continuation of U.S. patent application Ser. No.11/382,170, filed May 8, 2006, now U.S. Pat. No. 7,451,041, which claimspriority to U.S. Provisional Patent Application No. 60/678,847, filedMay 6, 2005, of the same title, the disclosure of which is herebyincorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention generally relates to data processing systems forvehicles, navigation and relative location. More specifically, thepresent invention is directed to a network-based navigation system foruse with a map database system that provides visual route guidanceinformation having virtual drive-thru advertisements integrated withactual imagery from along a physical route.

2. Background Art

Computerized route guidance and navigation systems are well known in theart. In-vehicle navigation systems, for example, rely on globalpositioning satellite (GPS) technology to provide current positionalinformation of a vehicle that is used to control the display of routespecific geographic or cartographic information maintained in a mapdatabase carried by the vehicle. In contrast, Internet-based navigationand mapping services like MapQuest®, Google Maps, Yahoo! Maps andMicrosoft MapPoint® rely on a user request for information about aparticular route or destination, instead of GPS data, and then use thisinformation to access various online map databases in order to providethe user with maps and travel instructions over the Internet.

While most in-vehicle navigation systems display the position of thevehicle or the desired route on conventional cartographic map displays,some in-vehicle navigation systems have been designed to provideadditional information beyond the conventional in-vehicle map display.For example, U.S. Pat. No. 5,115,398 describes an in-vehicle navigationsystem in which the navigation data is used to superimpose an indicationsignal on a real-time image of the roadway generated by aforward-looking video image pick-up unit in the vehicle. U.S. Pat. No.5,982,298 describes an in-vehicle navigation system with a userinterface that integrates highway-monitoring video with mapsinformation. U.S. Pat. No. 6,525,768 describes a similar arrangementutilizing a personal digital assistant, instead of a completelyin-vehicle based system. Similar kinds of additional information havealso been integrated into Internet-based navigation systems, as shownfor example, in U.S. Pat. Nos. 6,477,460 and 6,498,982.

Navigation systems have been developed that correlate aerialperspectives of a route (either actual or virtual) with map-basedinformation in order to provide the equivalent of a bird's-eye viewflyover of the route. U.S. Pat. No. 6,653,990 describes an in-vehicleversion of this approach. U.S. Publication Nos. 2004/0218910A1 and2002/0059207A1 describe Internet-based versions of this approach.

Navigation systems have also attempted to integrate driver perspectivesof a route (either actual or virtual) with map-based information inorder to provide the equivalent of a first-person drive-thru of theroute. U.S. Pat. No. 4,937,950 describes an early attempt to incorporatea three-dimensional presentation of an in-vehicle map-based navigationroute. U.S. Pat. No. 5,613,055 describes the use of driver-point-of-viewgenerated animation of a navigation route. U.S. Pat. Nos. 6,199,014 and6,351,710 describe the use of driver-point-of-view photographs for anInternet-based navigation system, while U.S. Pat. No. 6,182,011describes the same approach used in an in-vehicle navigation system. PCTPublication No. WO 99/39160A1 describes the use of firstperson-perspective video clips of a route to be navigated, such asdisplayed on a handheld device. Various techniques for acquiring videoand photographic images of a physical roadway from a driver-perspectiveare described in U.S. Pat. Nos. 5,633,946, 6,449,384, 6,453,056, and6,526,352, including the recognition of road signs along the roadway.

In-vehicle navigation systems have been developed that make use ofinformation alongside or near a roadway as part of the navigationsystem, as described, for example, in U.S. Pat. Nos. 6,671,615 and6,836,724. Other in-vehicle systems as described in U.S. Pat. Nos.5,664,948, 5,627,549, 6,542,822 and 6,714,860, and U.S. Publication No.2003/0006911 A1, for example, have been developed to present informationabout road signs and other forms of advertisements to the occupants of avehicle as the vehicle passes the vicinity of the road sign oradvertisement. One in-vehicle navigation system that provides augmentedinformation in a video display of the current route is described in U.S.Pat. No. 6,604,398.

Even though in-vehicle navigation systems have made use of additionalinformation from road signs and advertisements to augment or supplementthe actual experience as a vehicle is traveling along a roadway, therehave been very few similar approaches developed for other types ofapplications. One application where there has been the selectivereplacement of video images, and particularly billboard advertisingimages, has been in the context of broadcasting sporting events. U.S.Pat. Nos. 6,208,386, 6,292,227 and 6,384,471 describe various techniquesfor processing video streams acquired by stationary cameras in order toeffectively replace actual billboards in a stadium with prerecorded orcomputer generated advertising images. The techniques for selectivereplacement of a portion of a video image acquired by a stationarycamera are generally well known as described, for example, in U.S. Pat.Nos. 5,353,392, 5,491,517, 6,252,632, 6,381,362, 6,525,780, 6,774,908and 6,778,224 and PCT Publication No. WO 98/24242 A1. While thesetechniques are well developed with respect to the changing fields ofview for a stationary camera, these techniques generally have not beenapplied in the context of actual imagery captured by a moving cameratraversing along a physical roadway.

It would be desirable to provide a network-based navigation system foruse with a map database system that improved upon the current approachesfor using actual imagery from along a physical route in conjunction withthe navigation instructions for a given route.

SUMMARY OF THE INVENTION

The present invention is a network-based navigation system having a userinterface and a computer server system that can access a map database,an actual imagery database and an advertisement database in order toprovide visual route guidance information having virtual drive-thruadvertisements integrated with actual imagery from along a physicalroute. The user interface permits a user to submit a request fornavigation information, receive visual route guidance information inresponse to the request and initiate a drive-thru presentation of atleast a portion of the route. The computer server determines at leastone route from the map database based on the request from the user andprovides route guidance information to the user interface in response.The computer server also identifies actual imagery from the imagedatabase associated with the at least one route and selectively replacesat least one polygon region identified in the actual imagery associatedwith the at least one route with at least one advertisement from the addatabase to create a drive-thru presentation with at least one virtualadvertisement. In one embodiment, at least a portion of the drive-thrupresentation is presented to the user interface and the computer serverrecords a drive-thru charge for the virtual advertisements present inthe portion of the drive-thru presentation provided to the userinterface.

In practice, the vehicle acquiring the video imagery is equipped with asufficient number of cameras mounted on the vehicle which afterpost-processing preferably provide a 360 degree field of view around thelocation of the vehicle at any given instant of time. The image framesare tagged with absolute and/or relative geo-spatial coordinates. When auser requests a virtual drive through of a particular route, the imagescorresponding to the geographical coordinates of points on the route areretrieved and replayed with original, replaced or synthetic drive-thruadvertisements.

All the advertisements along the route are flagged as visited by theuser as a “drive thru” are played on the user interface. In oneembodiment, the “drive thru” provides a metric of the eyeballs that wereexposed to the advertisement and therefore a first rate billed to theadvertiser associated with each advertisement. Alternatively, the usermay “stop” along the route during a drive-thru and click on a particularadvertisement to be hyperlinked to the advertiser's website. The “clickthru” is recorded and provides a metric of the eyeballs attracted to aparticular advertisement and therefore of a second rate billed to theadvertiser for the click thru, as opposed to the drive thru.

The following figures are not drawn to scale and only detail a fewrepresentative embodiments of the present invention, more embodimentsand equivalents of the representative embodiments depicted herein areeasily ascertainable by persons of skill in the art.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of illustrating an image of an exemplaryroadway environment containing different objects of interest inaccordance with the present invention.

FIG. 2 is a schematic diagram of the network-based navigation systemhaving virtual drive-thru advertisements integrated with actual imageryfrom along a physical route

FIG. 3 shows a representation of a geo-referenced image N that containsa sign.

FIG. 4 shows a representation of a geo-referenced image N+1 thatcontains a sign.

FIG. 5 shows a representation of a geo-referenced image N+2 thatcontains a sign.

FIG. 6 shows a representation of a geo-referenced image N+3 thatcontains a sign.

FIG. 7 shows the definitions of the image bounding box and the signborder polygon for a sign completely within the camera's field of view.

FIG. 8 shows sign polygon definitions for a sign partially within thecamera's field of view.

FIG. 9 is an event flow diagram illustrating the operation of a systemaccording to one embodiment of the present invention; and

FIG. 10 is a flow chart illustrating the method of identifying ad-spotsaccording to one embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to FIG. 1, there is represented schematically a roadway 20.The roadway 20 has disposed on either side and in front of it, road sign22, an advertisement sign 10, interspersed along with natural landmark42 and manmade landmark 44. Advertisement sign 10 has a firstadvertisement 52 displayed on the sign.

A data acquisition vehicle (not shown) is driven along the roadway 20and is equipped with an image acquisition system. According to anembodiment of the present invention, the image acquisition systempreferably comprises six cameras positioned above roadway 20. Twocameras may be disposed on the right and the left side of the front ofvehicle respectively, cameras may also be disposed on the right and theleft side of the rear of vehicle respectively, and cameras may desirablybe disposed on the right and the left side of the vehicle respectively.In particular, other than views of the objects and scenery on theroadway ahead and behind the vehicle on the roadway, objects and sceneryin the lateral view of the vehicle are also acquired. The cameras may bepositioned inside of the vehicle and directed to the outside through thewindshield for example.

One of skill in the art will recognize that other configurations of thecameras may be utilized within the scope of the present invention. Inparticular, the configuration used may be the one disclosed in U.S. Pat.No. 6,453,056, entitled “Method and Apparatus for Generating a Databaseof Road Sign Images and Positions”, the disclosure of which isincorporated herein in its entirety. The objective of the imageacquisition system is to obtain images, either moving or still, digitalor photographic, that can be processed into a visual imagery databasecontaining images of the roadway 20 from a driver's perspective, thedriver's perspective being generally horizontally oriented and having afield of vision that extends generally laterally from the front, back orsides of the image acquisition vehicle. In a general embodiment of thepresent invention, the complete imagery generating passes generateimagery corresponding to a 360 degree field of view at each point or theimagery can be composed into a virtual 360 degree view at any pointalong the roadway after the images have been compiled/processed. Thecamera system disclosed above takes images of physical scenes andobjects along a roadway. The cameras are operably coupled to a computerprocessor and a database (not shown) as by electrical or fiber opticcables. The processing equipment may be local or remote.

The sequence of images are processed to generate image frames taggedwith meta data such as the location of the vehicle, the orientation andidentity of the camera, the speed of the vehicle, the environmentalconditions, the time and date of image acquisition and other data ofinterest. The image processing system detects objects in the imagestream and categorizes them into at least one or more of road signs,advertising signs, natural objects, and manmade landmarks. Theimage-processing unit also extracts other information from the imagestream from each camera such as, for example, the distances between thevehicle and the signs/landmarks in the image stream, thethree-dimensional configuration of the roadway, and other data that maybe of interest. Additionally, each image frame may be augmented withmeta data such as positional data locating the vehicle on a computerizedmap. This is achieved, for example, by coupling the system to a GPSsystem or by utilizing cell phone tower triangulation or other similarposition referencing techniques. A preferred embodiment of the methods,apparatus and systems utilized by the present invention are disclosed inU.S. Pat. No. 6,449,384, entitled “Method and Apparatus for RapidlyDetermining Whether a Digitized Image Frame Contains an Object ofInterest,” which is incorporated herewith in its entirety. One of skillin the art will recognize that the real-time images, such as forexample, video imagery, may be viewed, processed, analyzed or stored inany number of known processes, techniques and systems.

An effective method for gathering ground-based geo-referenced imagery ispracticed by attaching digital cameras and a positioning system to avehicle and operating the vehicle over roadways. As the vehicletraverses the roadways, the cameras periodically capture images that aretagged with positional information. Multiple cameras are typically usedin order to provide a rich user experience with the gathered imagery.All geo-referenced images can be uniquely identified by their run ID,camera number, and image number. The run ID refers to a grouping ofimages collected by a specific vehicle on a given date. Each image “run”will typically start with image number 0 and continue with sequentiallynumbered images throughout the run. Each camera on a vehicle willproduce sequentially numbered images for a given run. The image framedata depict roadside scenes as recorded from a vehicle navigating theroad. These scenes will generally contain information about signage andlandmarks that may be mined to characterize the signage and landmarks bytype, location, orientation, and condition.

FIG. 2 depicts an information flow diagram illustrating the processingof the image streams and/or image frames in a general embodiment of thepresent invention. A preferred embodiment of a method and apparatus forprocessing the objects is described in U.S. Pat. No. 6,363,161, entitled“System for Automatically Generating Database of Objects of Interest byAnalysis of Images Recorded by Moving Vehicle,” which is incorporatedherewith in its entirety.

In one embodiment, the present invention allows for the identificationof ad-spots within geo-referenced imagery by utilizing differentiablecharacteristics of the portions of the image frame that depict a roadsign, ad-spots, and other manmade and natural features. Thesedifferentiable characteristics are stored as highly compressed bitmappedfiles each linked to a discrete data structure containing, but notnecessarily limited to, one or more of the following memory fields:sign/landmark type, relative or absolute location of each sign/landmark,reference value for the recording camera, reference value for originalrecorded frame number for the bitmap of each recognized sign. Locationsmay be determined from a single image by determining depth informationfrom shading or size estimation. Alternatively, the position may bedetermined by mathematical methods such as triangulation. Cameras may becalibrated in the factory or may be estimated “in the field” by avariety of well-known techniques. Alternatively, distance-rangingequipment such as a laser can yield an accurate measurement of distance.In other embodiments, locations may be determined or derived fromanalysis of multiple images using any one or more of these techniques.

Output signal sets resulting from application of the present method to asegment of image frames can include a compendium of data about eachsign/landmark and bitmap records of each sign/landmark as recorded by acamera. Thus, records are created for image-portions that possess (andexhibit) detectable unique differentiable characteristics versus themajority of other image portions of a digitized image frame. In theexemplary sign-finding embodiment herein these differentiablecharacteristics are coined “sign-ness.” Based on the differentiablecharacteristics, or sign-ness, information regarding the type,classification, condition (linked bitmap image portion) and/or locationof road signs (and image-portions depicting the road signs) are rapidlyextracted from image frames. Those image frames that do not contain anappreciable level of sign-ness are categorized as background scenery.The image frames that contain an appreciable level of sign-ness arefurther processed and separated into two categories—road signs andpotential advertisements.

After the image stream has been processed, it is in effect decomposedinto component regions broadly classified according to the objectsencountered within. In one embodiment, the present invention utilizesseveral broad categories exemplified in FIG. 1—road signs 22,advertisement signs 10, natural landmarks 42, manmade landmarks 44 andpavement 20. Other broader generalizations or narrow categories may beemployed without digressing from the scope of the invention disclosedherein. Each shape is identified at least in regard to its attributessuch as shape, color, and orientation with respect to each of thecameras that imaged the object, absolute location, relative location,relation with respect to the pavement. Each of these attributes may betime dependent due to the travel speed of the vehicle. Database ofadvertising signs may include billboards, storefronts, bus stops,kiosks, signage as well as the typical road signs that warn, inform ordirect. A template database used to detect the presence of road signs ina video stream may be augmented with additional templates correspondingto each of the above advertisement signs. Alternately, image recognitionor other well-known methodologies may be utilized to decipher thecontents of non-road sign objects so as to classify them intoadvertisement and non-advertisement related categories.

In one embodiment, the present invention transforms frames of a digitalimage stream depicting roadside scenes using a set of filters that arelogically combined together with OR gates or combined algorithmicallyand each output is equally weighted, and that each operate quickly tocapture a differentiable characteristic of one or more signs/landmarksof interest. The image frames are typically coded to correspond to acamera number (if multiple cameras are used) and camera location data(i.e., absolute location via GPS or inertial coordinates if INS iscoupled to the camera of the camera-carrying vehicle). If the locationdata comprises a time/position database directly related to frame number(and camera information in a multi-camera imaging system) extremelyprecise location information is preferably derived using triangulationof at least two of the related “images” of a confirmed object (roadsign).

In one embodiment, the present invention handles partially obscuredsigns, skewed signs, poorly illuminated signs, signs only partiallypresent in an image frame, bent signs, and ignores all other informationpresent in the stream of digital frame data (preferably even the poststhat support the signs). Cataloging the location, direction the camerais facing, condition, orientation, and other attributes of objects suchas advertisement, road signs and landmarks to assist navigation ofvehicles can be successfully completed by implementing the inventivemethod described herein upon a series of images of said objects. In ageneral embodiment, the present invention can quickly and accuratelydistill arbitrary/artificial objects disposed in natural settings andexcept for confirming at least one characteristic of the object (e.g.,color, linear shape, aspect ratio, etc.), the invention operatessuccessfully without benefit of preexisting knowledge about the fullshape, actual condition, or precise color of the actual object.

The image frames that contain an appreciable level of sign-ness andcategorized as potential advertisements are further processed toidentify the region of the image covered by the advertisement. Eachregion categorized as an advertisement is assigned a unique identifierthe first time it is encountered in a video stream. Typically, thespecific region will persist, albeit at different orientations, in asuccession of frames corresponding to a travel-rate of the camera(s)along a specific route. All the frames that contain the regioncorrespond to one unit of advertisement space or an ad-spot in thedatabase. The ad-spots will be identified within all images so theplacement of the original ads can be tracked. All ad-spots that displaythe same physical ad-spot will be indexed together in the ad-spotdatabase. Metrics that can be determined for each ad-spot include, butare not limited to, frequency of a message, frequency of an advertiser,number of frames that contain the same physical ad-spot, relative sizeof the ad-spot compared to the image size, type of road from which thead-spot appears, density of traffic on the selected roadway, etc. Thesead-spots are available for leasing by advertisers. An existingadvertisement can be left as-is, replaced by an alternate advertisementor by background scenery to effectively erase the advertisement spacefrom the frames. In an alternate embodiment, the images may be replacedwith synthetic data. Images of real-world scenes may have objectsremoved in order to improve the viewing experience. In addition, theimages, at the user request, can be altered to fit the current lightingand weather conditions.

FIG. 2 schematically illustrates the operative principles governingimage acquisition in a first embodiment of the instant invention, whichincludes acquiring, processing and storing a series of images along atypical route traversed by the vehicle. Notably, the image sequencesalong a particular roadway are related in that they are images of thephysical scenery associated with a particular roadway, acquired by aknown camera configuration, by a vehicle traveling along the roadwayunder a particular set of environmental conditions, and at a specifiedrate of speed.

The present invention may be implemented in a single microprocessorapparatus, within a single computer having multiple processors, amongseveral locally-networked processors (i.e., an intranet), or via aglobal network of processors (i.e., the Internet and similar). Portionsof individual image frames exhibiting an appreciable level ofpre-selected differentiable characteristics of desired objects areextracted from a sequence of image frame data and portions of theindividual frames (and correlating data thereto) are used to confirmthat a set of several “images” in fact represent a single “object” of aclass of objects. These pre-selected differentiable characteristiccriteria are chosen from among a wide variety of detectablecharacteristics including color characteristics (color-pairs and colorset memberships), edge characteristics, symmetry, convexity, lack of 3Dvolume, number and orientation of side edges, characteristic cornerangles, frequency, and texture characteristics displayed by the2-dimensional (2D) images so that said objects can be rapidly andaccurately recognized. In one embodiment, the differentiablecharacteristics are chosen with regard to anticipated camera directionrelative to anticipated object orientation so that needless processingoverhead is avoided in attempting to extract features andcharacteristics likely not present in a given image frame set from aknown camera orientation. Similarly, in the event that a scanningrecording device, or devices, are utilized to record objects populatinga landscape, area, or other space the extraction devices can bepreferably applied only to those frames that likely will exhibitappreciable levels of an extracted feature or characteristic.

In one embodiment, the inventive system taught herein is applied toimage frames and an output signal set of location, type, condition, andclassification of each identified sign/landmark is produced and linkedto at least one bitmap image of the sign/landmark. The output signal setand bitmap record(s) are thus available for later scrutiny, evaluation,processing, and archiving. Of course, pre-filtering or conditioning theimage frames may increase the viability of practicing the presentinvention. Some examples include color calibration, color densityconsiderations, video filtering during image capture, etc.

The envisioned system can be built for entire geographic entities fromcities to countries to the entire planet. Due to the enormity of thedata size and the need for rapid geospatial queries, the indexing can begeospatially based. The geospatial indexing also allows for ease ofupdating of the system by multiple traversals of a location regardlessif they are years apart in time. The resulting symmetric decrease instorage size is also a strong feature of the geospatial representationmethod.

Viewpoint and viewing angle are best represented by the geospatialindexing process as the computations are then reduced to simpletrigonometry versus complex database queries. This method of storageeliminates any difficulties with occluded or bent objects as their“true” extents can be represented even if not visible from the imagery.

The present invention may be practiced with imaging systems ranging frommonochromatic visible wavelength camera/film combinations to full colorspectrum visible wavelength camera/memory combinations to ultraviolet,near infrared, or infrared imaging systems, so long as basic criteriaare present: object differentiability from its immediate milieu or rangedata.

Image frames that are temporally related—i.e. taken at or about aparticular instant of time t1—may be “stitched” together to obtain acomposite image of the environment surrounding the vehicle at theinstant of time. Successive composite images representing progression ofthe vehicle along the roadway, can be further processed to obtain acomposite real-time video of the scene unfolding as would appear to adriver of the vehicle viewing the roadway along a specified driver'sline-of-sight from within the vehicle. Those of ordinary skill in theart of machine vision and image processing will recognize that there arenumerous schemes for performing such transformations.

For example, U.S. Pat. No. 5,649,032, which is incorporated herein byreference, discloses methods for automatically generating a mosaic froma sequence of images. The invention can construct both, dynamic (i.e.changing over time) and static (no change with time) mosaics. U.S. Pat.No. 6,512,857, which is incorporated herein by reference, discloses asystem and method for accurately mapping between camera-coordinates andgeo-coordinates, called geo-spatial registration. The method utilizesthe imagery and terrain information contained in the geo-spatialdatabase to precisely align the reference imagery with input imagery,such as dynamically generated video images or video mosaics, and thusachieves a high accuracy identification of locations within the scene.The geo-spatial reference database generally contains a substantialamount of reference imagery as well as scene annotation information andobject identification information. When a sensor, such as a videocamera, images a scene contained in the geo-spatial database, the systemrecalls a reference image pertaining to the imaged scene. This referenceimage is aligned very accurately with the sensor's images using aparametric transformation. Thereafter, other information that isassociated with the reference image can easily be overlaid upon orotherwise associated with the sensor imagery.

A specific embodiment of the present invention is illustrated in FIGS.3, 4, 5, 6, and 7 which shows four consecutive geo-referenced imagesgathered from a roadway. A sign 10 next to the actual roadway 20 appearsin all four views 30, 40, 50, and 60. As described below, there iscreated a database model that describes the locations of these signviews 30, 40, 50, 60 in a way that will allow the tracking andreplacement of the sign 10 with another advertisement.

FIG. 7 shows the locations of the four points of the sign borderpolygon: x₀,y₀ 100; x₁,y₁ 110; x₂,y₂ 120; x₃,y₃ 130. If it is assumedthat the sign is only seen from the four views 30, 40, 50, 60 shown, thedatabase model for the sign 10 will be as shown in Table 1 below:

TABLE 1 Sign Run Cam. Image# S₀ S₁ S₂ S₃ M K 0 N x₀, y₀ x₁, y₁ x₂, y₂x₃, y₃ M K 0 N + 1 x₀, y₀ x₁, y₁ x₂, y₂ x₃, y₃ M K 0 N + 2 x₀, y₀ x₁, y₁x₂, y₂ x₃, y₃ M K 0 N + 3 x₀, y₀ x₁, y₁ x₂, y₂ x₃, y₃Each row in the database corresponds to a distinct image 30, 40, 50, and60 that contains the ad-spot. S₀ through S₃ define the corner points forthe four-sided polygon that makes up the ad-spot border. Each image 30,40, 50, 60 will have a unique S₀ through S₃ 100, 110, 120, 130 since thead-spot will inhabit a different location within each frame.

For more complete user experiences, image collection vehicles willtypically implement multiple cameras as has been noted above. If thevehicle in the example above has a second camera aimed to the right ofthe forward-facing camera, the sign will possibly be seen in one or moreimages captured with this second camera. With the second camera, thead-spot database for this sign would be as shown in Table 2 below:

TABLE 2 Sign Run Cam. Image# S₀ S₁ S₂ S₃ M K 0 N x₀, y₀ x₁, y₁ x₂, y₂x₃, y₃ M K 0 N + 1 x₀, y₀ x₁, y₁ x₂, y₂ x₃, y₃ M K 0 N + 2 x₀, y₀ x₁, y₁x₂, y₂ x₃, y₃ M K 0 N + 3 x₀, y₀ x₁, y₁ x₂, y₂ x₃, y₃ M K 1 N + 3 x₀, y₀x₁, y₁ x₂, y₂ x₃, y₃ M K 1 N + 4 x₀, y₀ x₁, y₁ x₂, y₂ x₃, y₃

Many image frames will contain partial views of signs since a portion ofthe sign will typically be outside the camera's field of view. In thesecases, one or more of the points S₀ through S₃ 100, 110, 120, 130 willspecify pixels that are outside the image frame. FIG. 8 introduces thepoints I₀ through I₃ 180, 190, 200, and 210 which is the image boundingbox that defines the area within the image frame onto which the partialnew advertisement should be placed. The overlay software will computethe portion of the desired image that needs to be clipped and placedwithin the I₀ through I₃ polygon. For ad-spots that are entirely withinthe image frame, S₀ through S₃ 100, 110, 120, and 130 will be identicalto I₀ through I₃ 180, 190, 200, 210. With the inclusion of partial signswithin the image frames, the ad-spot database would be as shown in Table3 below:

TABLE 3 Sign Run Cam. Image# S₀ S₁ S₂ S₃ I₀ I₁ I₂ I₃ M K 0 N x₀, y₀ x₁,y₁ x₂, y₂ x₃, y₃ x₄, y₄ x₅, y₅ x₆, y₆ x₇, y₇ M K 0 N + 1 x₀, y₀ x₁, y₁x₂, y₂ x₃, y₃ x₄, y₄ x₅, y₅ x₆, y₆ x₇, y₇ M K 0 N + 2 x₀, y₀ x₁, y₁ x₂,y₂ x₃, y₃ x₄, y₄ x₅, y₅ x₆, y₆ x₇, y₇ M K 0 N + 3 x₀, y₀ x₁, y₁ x₂, y₂x₃, y₃ x₄, y₄ x₅, y₅ x₆, y₆ x₇, y₇ M K 1 N + 3 x₀, y₀ x₁, y₁ x₂, y₂ x₃,y₃ x₄, y₄ x₅, y₅ x₆, y₆ x₇, y₇ M K 1 N + 4 x₀, y₀ x₁, y₁ x₂, y₂ x₃, y₃x₄, y₄ x₅, y₅ x₆, y₆ x₇, y₇

FIGS. 3, 4, 5, 6, and 7 illustrate an embodiment of the presentinvention wherein the ad-spot is a man-made sign along the roadway. Forpurposes of this invention, an ad-spot can be defined as any geospatiallocation that can reasonably accommodate an advertisement inserted intothe image stream including, but not limited to store windows, buildingfaces, road surfaces, lake surfaces, open fields, mountain tops, rockfaces, clouds, trees, car windows, etc.

In a general embodiment of the present invention, illustrated in theevent flow diagram of FIG. 9, the data collected in a first context,such as for example, the context of an ad-spot, is leveraged to generateadditional revenue streams. As discussed above, the processing of theimage stream is expanded to retrieve information about regions thatlikely are advertisement signs or candidates for placement ofadvertisements. Each of these regions corresponds to a virtual ad-spaceand like its real property ad-space counterpart the virtual ad-space maybe leased, rented, sold or otherwise transacted as part of revenuegenerating operations. One of the steps in the transaction is valuationof the virtual ad-space. In this regard, each object in the image streamis further linked to other non-navigation related attributes. Forexample, each object may be linked to a database of addresses, proximityto business centers, demographics, local attractions, traffic density asa function of the time of day and other decisional criteria employed invaluation of an real-property ad-space.

In FIG. 9, another embodiment of the present invention is shown in whichthe input data is heterogeneous, in that a simple list of locations 210having geo-spatial coordinates 220 or a complete route 230 made from adifferent router and map database may be sent to the system. In the caseof a route 230 being received, the system requires at least theintersection locations to be provided. Better quality matching willensue as more data is supplied. Better quality output will be realizedwhen the inputs to the system include full route points, names ofstreets, and turn direction information. In one embodiment, input datais sent to a heavily hinted conflation engine 240 that matches thedesired route data to the system's map data 260. Map data 260 maycomprise a map database, weather data, and traffic data, and map data260 is in communication with a routing engine 250. This is segment-levelmatching, so the size difference between similar segments in thedifferent databases is not important. Thus, a route 270 is generated ofsystem data with references to the positions that are intersections andturns in the original route 220.

Once route data for a route has been created, the route 270 goes throughthe imagery matching process 290. The database can be sampled in avariety of ways from highest possible image density where all availableimagery is provided to just cross streets or intersections with routechanges (turns). In one embodiment, at each identified position for arequested image the image database 300 is queried for the best match280. Based on the match quality parameters of the request there may ormay not be a successful match. Match quality parameters include, but arenot limited to DISTANCE, FIELD OF VIEW, AGE, QUALITY and 3D. The imagesare graded and the best is selected for usage. At the same time ageospatial query is performed on the other databases such as Ad Database310 and Service Area database 320, with the query parameters includingdemographic, time of day, service type, buying patterns and potentialspecific buyer preferences (for example, KFC versus Boston Chicken). Adisplayable route 330 is then generated, including imagery andapplicable advertising.

In one embodiment, the geospatial query for the Ad Database 310 returnsthe position of ad areas, and the other databases return the appropriateads. The images can then have the ads inserted, and the appropriatefiltering is done to the images. The filtering may correct for light,weather, leaf on, obscuring objects, and other external factors. Thefiltering will produce an image of the correct and format for theintended user regardless of its relationship to the original imagery. Inone embodiment, the route is updateable, by either reprocessing with anew start node and the rest of the route or by just routing back to theroute.

In one approach, modeled in part upon the disclosure of U.S. PublicationNo. 2005/0021472 A1, entitled “Transactions in Virtual Property,” andincorporated in its entirety by reference herein, the ad-spacetransactions may involve end users, advertisers or content site owners,city operators, and a universe operator. End users, who are also termedviewers herein because they may request a destination or a route to adestination or specify a route in the database of roads either forpreviewing a trip by driving a virtual route on a generic device such asa computer, a hand-held such as a personal digital assistant (PDA), amessaging device, a cellular phone, a game console, interactive TV orthe user may request the display on a specialized navigation device, theuser may be stationed at a remote location from the requested route orthe user may be traveling to the destination along a section of arequested route. In all cases, the present invention will guide the uservia the roads in the database to the destination. The guidance takes theform of continually recreating the 3-D imagery surrounding each locationalong the virtual road as the vehicle travels down the roadway. Theimagery displayed is keyed to a line of sight of the driver of theactual or virtual vehicle. Additionally, turn-by-turn instructions maybe superimposed on the evolving roadway.

In one embodiment of the instant invention, the user interacts with aremote repository of information through a user interface connected to anetwork. The term “network” is used here in a very broad sense to denotean interconnected set of nodes at which one or more resources requiredto effectuate the present invention may be located. A network therefore,may be an intranet, the Internet, a wireless network or any othercommunication network. The user interface may be a personal computer, ahandheld, a cell phone, a dedicated video terminal, an interactive TVmonitor or any other device equipped to accept user input and displaythe results obtained from a locally or remotely implementedroute/advertisement related computation.

As illustrated in FIG. 2, the user interacts with a remote server whichis in turn connected to a map database, an image database, and anadvertisement database. The server or a computing machine connected tothe server utilizes code resident on it to process the user request inconjunction with the information in the databases to generate routerelated imagery and the attendant advertisement that are sent to theuser interface for display. In a special embodiment of the presentinvention, the repository also consists of a database that storespersonal demographic/buying preferences. Operationally, the location ofthe user, the time of day or other user-preference modifiers will beused to select the advertisements to which the user will be exposed.

In another embodiment, at least one of the databases discussed above islocated remotely from the user so as to be accessible over the network,whereas the rest of the databases are located locally, for example, on acell phone or handheld device from which the user is interacting with anetwork. In an exemplary embodiment, a cell phone that has an imagedisplay with the routing data and personal demographic/buyingpreferences stored locally. Imagery and signage position data are sentbased on requests from the phone, which has positional determination inaccordance with any of the previously identified techniques (Pull DataBased on Position). Alternately, the imagery and the map data may resideon the local device or be read into the local device using an externalrepository of information. The positional data system will cooperatewith the personalized demographic data and the advertising databases todetermine, based on a plurality of factors such asposition/time/previous task(s), the advertisements that will bedisplayed.

One of skill in the art will recognize that every recorded image framedoes not have to be processed or displayed, as this will incur hugeoverhead in terms of computational, storage and access costs. Initialimages that are acquired in analog video format may be digitized. Thedigitized images may be converted to a JPEG or MPEG format data streamto conserve bandwidth, or may be presented as a series or sequence ofstill images.

As illustrated in FIGS. 1, 3-7, certain locations 10 that carrypre-defined content 52, and therefore tagged as advertisement signs inthe pre-processing discussed above are displayed as is at the same or adifferent location. If a location has been “sold” to an advertiser otherthan the one advertising at the location in the physical world, theoriginal real-world ad sign is replaced with a different advertisement.Locations that are not used for advertisement purposes in the real worldmay be suited to carry an ad in the virtual world. The advertisementand/or the entire environment viewed by the driver may be synthetic inthat it is artificially created.

Methods and systems to replace advertisements in a video stream are wellknown in the art. For example, U.S. Pat. No. 6,381,362, entitled “Methodand Apparatus for Including Virtual Ads in Video Presentations” andincorporated by reference in its entirety herein, teaches a system forpresenting a virtual advertisement or other virtual content in asequences of frames of a video presentation. The system utilizes dataidentifying a candidate region in the sequence of frames, a pixelsignature for the region, the signature comprising a range of R, G, Bvalues for pixels based on a standard deviation from an average value,and stored virtual content, such as an advertisement, to be implanted inthe candidate region or regions. In operation, as frames having thecandidate region are processed by an integration system, the alternativevirtual content is substituted by default for the candidate region,except if a pixel value in the region falls outside the pixel signaturerange. The result is an ad that appears to be on a surface in the videopresentation, as images that come between the ad and the viewer occludethe advertisement. While it is conceivable to utilize non-polygon shapesand spaces in a video stream for selectively replacement withadvertisements, the preferred embodiment of the present inventionutilizes polygon shapes having defined boundary edges (e.g., squares,rectangles, etc) to minimize the amount of transformation that isrequired to map an advertisement from the ad database into theidentified replacement region in the actual imagery.

For purposes of the present invention, the Ad Database refers to a setof information that can be inserted into select locations in the images.The “click-through” functionality of the present invention may or maynot rely on the Ad Database. Actual advertisements displayed in theactual imagery can be alternately identified and their presence in theimagery can be monetized by charging the original advertiser forclick-thrus and/or eyeballs that view the imagery over the network.

In one embodiment, the present invention provides for a counter that isincremented each time a user navigates past a virtual ad. In a separateembodiment, the advertisement may be hyperlinked to allow a user toclick on the link and follow it to the merchant's website or place acall to a brick-and-mortar facility. The number of clicks thru anadvertisement or the trips down a segment of a roadway where anadvertisement is displayed on the virtual roadway is used as a metric todetermine the rate the advertiser is charged for the virtual ad. In analternate embodiment, displayed ads can be tallied and data about viewscan be stored locally and/or sent for billing determination. Additionalrevenue streams are generated by providing a gateway for a potentialclient to navigate to or hyperlink to a particular merchant'sbrick-and-mortar operation or website respectively. For example, U.S.Pat. No. 6,820,277, entitled “Advertising Management System for DigitalVideo Streams,” incorporated herein in its entirety, teaches anadvertisement management system (AMS) in accordance with the principlesthat consists of one or more subsystems which allow for thecharacterization of the advertisement, determination of advertisingopportunities (avails), characterization of the subscriber, correlationof the advertisement with a subscriber or group of subscribers, and saleof the advertisement, either through a traditional placement (sale), anInternet based sale, or an Internet based auction. The disclosure ofthis patent may be instructive as to the various ways in which thepresent invention may be mined for advertisement related revenuegeneration.

The complete disclosures of the patents, patent applications andpublications cited herein are incorporated by reference in theirentirety as if each were individually incorporated. Variousmodifications and alterations to this invention will become apparent tothose skilled in the art without departing from the scope and spirit ofthis invention. It should be understood that this invention is notintended to be unduly limited by the illustrative embodiments andexamples set forth herein and that such examples and embodiments arepresented by way of example only with the scope of the inventionintended to be limited only by the claims set forth herein.

1. A method of identifying ad-spots within geo-referenced imagery byutilizing sign-ness characteristics of portions of an image frame thatdepict suitable advertisement locations, the method comprising:providing an image database containing actual imagery captured from atleast one camera traversed along a physical route, the actual imageryhaving a generally horizontally oriented perspective; storing thesign-ness characteristics in a database as discrete data structurescontaining one or more information fields relating to the suitableadvertisement locations; creating records for image portions thatexhibit detectable unique sign-ness; extracting image portions depictingsuitable ad-spots from image frames having an appreciable level ofsign-ness; categorizing image frames with an unappreciable level ofsign-ness as background imagery; processing image frames containing anappreciable level of sign-ness, such that the suitable advertisementlocations are categorized as road signs or potential advertisements; andidentifying a region of an image frame containing a potentialadvertisement and assigning a unique identifier to the region, whereinmultiple successive image frames containing the region comprises anad-spot.
 2. The method of claim 1, further comprising conditioning animage frame to increase the desirability of an ad-spot.
 3. The method ofclaim 1, further comprising linking an ad-spot to non-navigation relatedattributes.
 4. The method of claim 1, further comprising determiningmetrics for the ad-spot.
 5. The method of claim 1, further comprisingmaking the ad-spot available to potential advertisers for leasing.
 6. Asystem for the identification of ad-spots within geo-referenced imageryby utilizing sign-ness characteristics of portions of an image framethat depict suitable advertisement locations, the system comprising: animage database containing actual imagery captured from at least onecamera traversed along a physical route, the actual imagery having agenerally horizontally oriented perspective; a database containing thesign-ness characteristics as discrete data structures containing one ormore information fields relating to the suitable advertisementlocations; means for creating records for image portions that exhibitdetectable unique sign-ness; means for extracting image portionsdepicting suitable ad-spots from image frames having an appreciablelevel of sign-ness; means for categorizing image frames with anunappreciable level of sign-ness as background imagery; means forprocessing image frames containing an appreciable level of sign-ness,such that the suitable advertisement locations are categorized as roadsigns or potential advertisements; and means for identifying a region ofan image frame containing a potential advertisement and assigning aunique identifier to the region, wherein multiple successive imageframes containing the region comprises an ad-spot.